Lecture Notes on Statistical Learning Theory

Problem setting. In statistical learning theory (SLT), the goal is to find a classifier g : Rd → {0, 1}, predicting the correct class y of an observation x ∈ Rd, based on data (x1, y1), . . . , (xn, yn). Clearly, we cannot learn any reasonable classifier, if no assumption is imposed on the relationship between the data and the test observation (x, y). To this end, we assume in SLT that the data pairs Dn := (xi, yi) n i=1 and the test observation (x, y) are independently drawn from one and the same probability distribution P. Correspondingly, we denote the random variables associated to (xi, yi) and (x, y) by capital letters, i.e., (Xi, Yi) and (X,Y ), respectively. Thus a classifier errs if g(X) 6= Y so that L(g) := P(g(X) 6= Y |Dn) is the probability of error of g.